1. Field of the Invention
The present invention relates to an image processor, an image processing method, and a program. More particularly, the present invention relates to an image processor, an image processing method, and a program, which are suitable for use when converting an input image to an image with a higher resolution.
2. Description of the Related Art
In the related art, a method called super-resolution is known as a method of obtaining an image with a higher resolution from an input image (for example, see Japanese Unexamined Patent Application Publication No. 2008-140012)
For example, according to a technique called super-resolution back-projection, a consecutive frame image such as a moving image, which is input to an image processor 11, is converted to an image with a higher resolution and output, as illustrated in FIG. 1. In the following description, the image with a lower resolution being input to the image processor 11 will be referred to as an LR (low resolution) image, and the image with a higher resolution being output from the image processor 11 will be referred to as an SR (super resolution) image.
The input LR image is up-sampled to an image with the same resolution as the SR image by an up-sampler 21, and the up-sampled image is supplied to a motion vector detector 22, a mask generator 23, and a mixer 24. The image processor 11 includes a buffer 25 storing an SR image which is obtained from an LR image of a previous frame immediately before the current frame being processed at the current point of time. The SR image supplied to the buffer 25 is stored only for one frame period and is supplied to the motion vector detector 22 and a motion compensator 26.
The motion vector detector 22 detects a motion vector from the LR image supplied from the up-sampler 21 and the SR image supplied from the buffer 25. The motion compensator 26 performs motion compensation using the motion vector supplied from the motion vector detector 22 and the SR image supplied from the buffer 25. Specifically, the motion compensator 26 predicts an SR image obtained from the LR image of the current frame using the SR image of the previous frame immediately before the current frame being processed and supplies the image obtained thus to the mask generator 23 and the mixer 24 as a prediction image.
The mask generator 23 generates a motion mask using the LR image supplied from the up-sampler 21 and the prediction image supplied from the motion compensator 26. The motion mask is information used for specifying a region of the LR image in which a moving subject is displayed, and which is generated by calculating a difference between the LR image and the prediction image.
The mixer 24 combines the prediction image supplied from the motion compensator 26 and the LR image supplied from the up-sampler 21 using the motion mask supplied from the mask generator 23 and supplies a combined image obtained thus to a down-sampler 27 and an adder 28. Specifically, the LR image and the prediction image are subjected to weighted addition with weighting factors determined by the motion mask, thus obtaining the combined image. When generating the combined image, the weighting factors are determined such that the contribution ratio of the LR image becomes larger in the region where a motion occurs, whereby image quality deterioration of the SR image resulting from prediction errors occurring in the region with a motion is suppressed.
The combined image obtained thus is down-sampled by the down-sampler 27, and a reduced image obtained thus is supplied to a subtractor 29. The reduced image is an image with the same resolution as the LR image.
The subtractor 29 generates a differential image by calculating a difference between the LR image supplied to the image processor 11 and the reduced image. The up-sampler 30 up-samples the differential image to an image with the same resolution as the SR image, thus obtaining an enlarged image. Subsequently, an adder 28 adds the enlarged image and the combined image, outputs an image obtained thus as an SR image of the current frame, and supplies the SR image to the buffer 25 to be stored therein.
In the image processor 11, as illustrated in FIG. 2, a combined image P11 which is obtained by predicting the SR image of the current frame is down-sampled to obtain a reduced image P12, and a difference between the reduced image P12 and an LR image P13 is calculated to obtain a differential image P14. The differential image P14 is an image which is indicative of an error in the reduced image P12 used as the LR image of the current frame obtained by prediction with respect to the LR image P13 of the current frame. That is to say, the differential image P14 can be said to be an error in the combined image P11 used as the predicted SR image with respect to a correct SR image of the current frame which should have been obtained if there was no error.
Therefore, by up-sampling the reduced image P14 and adding a signal obtained thus to the combined image P11, an image which is further similar to the correct SR image which should be obtained. That is, the obtained SR image will become an image in which the LR image is more faithfully enlarged without any image quality deterioration.
As described above, according to the back-projection, an enlarged image which is indicative of the calculated error is added to the combined image obtained by prediction, thus obtaining the SR image.